Simultaneous speech-to-text translation (Simul-S2TT) aims to translate speech into target text in real time, outputting translations while receiving source speech input, rather than waiting for the entire utterance to be spoken. Simul-S2TT research often modifies model architectures to implement read-write strategies. However, with the rise of large audio-language models (LALMs), a key challenge is how to directly activate Simul-S2TT capabilities in base models without additional architectural changes. In this paper, we introduce {\bf Simul}taneous {\bf S}elf-{\bf A}ugmentation ({\bf SimulSA}), a strategy that utilizes LALMs' inherent capabilities to obtain simultaneous data by randomly truncating speech and constructing partially aligned translation. By incorporating them into offline SFT data, SimulSA effectively bridges the distribution gap between offline translation during pretraining and simultaneous translation during inference. Experimental results demonstrate that augmenting only about {\bf 1\%} of the simultaneous data, compared to the full offline SFT data, can significantly activate LALMs' Simul-S2TT capabilities without modifications to model architecture or decoding strategy.

When the complete source sentence is provided, Large Language Models (LLMs) perform excellently in offline machine translation even with a simple prompt "Translate the following sentence from [src lang] into [tgt lang]:". However, in many real scenarios, the source tokens arrive in a streaming manner and simultaneous machine translation (SiMT) is required, then the efficiency and performance of decoder-only LLMs are significantly limited by their auto-regressive nature. To enable LLMs to achieve high-quality SiMT as efficiently as offline translation, we propose a novel paradigm that includes constructing supervised fine-tuning (SFT) data for SiMT, along with new training and inference strategies. To replicate the token input/output stream in SiMT, the source and target tokens are rearranged into an interleaved sequence, separated by special tokens according to varying latency requirements. This enables powerful LLMs to learn read and write operations adaptively, based on varying latency prompts, while still maintaining efficient auto-regressive decoding. Experimental results show that, even with limited SFT data, our approach achieves state-of-the-art performance across various SiMT benchmarks, and preserves the original abilities of offline translation. Moreover, our approach generalizes well to document-level SiMT setting without requiring specific fine-tuning, even beyond the offline translation model.

Simultaneous interpretation (SI), the translation of one language to another in real time, starts translation before the original speech has finished. Its evaluation needs to consider both latency and quality. This trade-off is challenging especially for distant word order language pairs such as English and Japanese. To handle this word order gap, interpreters maintain the word order of the source language as much as possible to keep up with original language to minimize its latency while maintaining its quality, whereas in translation reordering happens to keep fluency in the target language. This means outputs synchronized with the source language are desirable based on the real SI situation, and it's a key for further progress in computational SI and simultaneous machine translation (SiMT). In this work, we propose an automatic evaluation metric for SI and SiMT focusing on word order synchronization. Our evaluation metric is based on rank correlation coefficients, leveraging cross-lingual pre-trained language models. Our experimental results on NAIST-SIC-Aligned and JNPC showed our metrics' effectiveness to measure word order synchronization between source and target language.

Simultaneous speech translation (SimulST) translates partial speech inputs incrementally. Although the monotonic correspondence between input and output is preferable for smaller latency, it is not the case for distant language pairs such as English and Japanese. A prospective approach to this problem is to mimic simultaneous interpretation (SI) using SI data to train a SimulST model. However, the size of such SI data is limited, so the SI data should be used together with ordinary bilingual data whose translations are given in offline. In this paper, we propose an effective way to train a SimulST model using mixed data of SI and offline. The proposed method trains a single model using the mixed data with style tags that tell the model to generate SI- or offline-style outputs. Experiment results show improvements of BLEURT in different latency ranges, and our analyses revealed the proposed model generates SI-style outputs more than the baseline.

Mozilla has created a translation plugin for Firefox that works offline. Firefox Translations will need to download some files the first time you convert text in a specific language. However, it will be able to use your system's resources to handle the translation, rather than sending the information to a data center for cloud processing. The plugin emerged as a result of Mozilla's work with the European Union-funded Project Bergamot. Others involved include the University of Edinburgh, Charles University, University of Sheffield and University of Tartu.

Mozilla has added an official translation tool to Firefox that doesn't rely on cloud processing to do its work, instead performing the machine learning-based process right on your own computer. It's a huge step forward for a popular service tied strongly to giants like Google and Microsoft. The translation tool, called Firefox Translations, can be added to your browser here. It will need to download some resources the first time it translates a language, and presumably it may download improved models if needed, but the actual translation work is done by your computer, not in a datacenter a couple hundred miles away. This is important not because a lot of people need to translate in their browsers while offline -- like screen door for a submarine, it's not really a use case that makes sense.

Currently, when the Google Translate apps for iOS and Android has access to the internet, its translations are far superior to those it produces when it's offline. That's because the offline translations are phrase-based, meaning they use an older machine translation technique than the machine learning-powered systems in the cloud that the app has access to when it's online. Google is now rolling out offline Neural Machine Translation (NMT) support for 59 languages in the Translate apps. Today, only a small number of users will see the updated offline translations, but it will roll out to all users within the next few weeks. The list of supported languages consists of a wide range of languages.

Neural machine translations (NMT) in the Google Translate application now works offline on both iPhone and Android, product manager Julie Cattiau announced in a blog post on Tuesday. Prior to this update, translations in the app were phrase-based, meaning sentences would be translated in chunks. The update will roll out in the next few weeks, according to the post. The offline functionality could be useful for international business travelers, especially those who travel to regions with spotty Wi-Fi or poor signal. The NMT functionality could also ease communication due to its more accurate translations.

Chances are you mostly need a translator app on your phone while you are traveling. But that's also when you are most likely to not have any connectivity. While most translation apps still work when they are offline, they can't use the sophisticated -- and computationally intense -- machine learning algorithms in the cloud that typically power them. Until now, that was also the case for the Microsoft Translator app on Amazon Fire, Android and iOS, but starting today, the app will actually run a slightly modified neural translation when offline (though iOS users may still have to wait a few days, as the update still has to be approved by Apple). What's interesting about this is that Microsoft is able to do this on virtually any modern phone and that there is no need for a custom AI chip in them.

Microsoft is delivering the world's first fully neural on device translations in the Microsoft Translator app for Android, customized for the Huawei Mate 10 series. Microsoft achieved this breakthrough by partnering with Huawei to customize Microsoft's new neural technology for Huawei's new NPU (Neural Processing Unit) hardware. This results in dramatically better and faster offline translations as compared to existing offline packs. The Microsoft Translator app with these capabilities comes pre-installed on Huawei Mate 10 devices allowing every Mate 10 user to have native access to online quality level translations even when they are not connected to the Internet. Until now, due to the computational requirements of neural machine translation, it was not possible to do full Neural Machine Translation (NMT) on-device.